Debriefing systems and methods for retrieving and presenting multiple datastreams with time indication marks in time synchronism

ABSTRACT

Event debriefing systems and methods are provided for recording a plurality of datastreams related to an event. An observer of the event may create time indication marks that facilitate recalling, during debriefing, what transpired at any particular point in time during the event. The recorded datastreams may be retrieved and presented beginning at marked time indications and continued chronologically thereafter in time synchronism.

This application is a continuation of Ser. No. 08/614,337 filed Mar. 12,1996, now U.S. Pat. No. 5,826,206.

FIELD OF THE INVENTION

The present invention relates generally to event recording systems, andmore particularly to systems for debriefing event participants.

BACKGROUND OF THE INVENTION

It is well known throughout the airline industry that flight crew erroris a contributing factor in a large number of aircraft accidents.However, many of the errors made by multi-person flight crews have verylittle to do with the technical aspects of flying an aircraft. Instead,a large number of these errors are related to poor group decisionmaking, ineffective communication, inadequate leadership, and poor taskmanagement (collectively referred to as “human factors” errors) amongthe flight crew.

Flight simulators were developed in the late 1960's and early 1970's toincrease flight crew proficiency in the technical aspects of flying.Flight simulators are computer controlled devices which simulate actualflying conditions of an aircraft and train flight crews to deal with avariety of situations before they actually encounter them in a realaircraft. Early on, the airline industry recognized the importance ofincreasing flight crew proficiency with respect to eliminating humanfactors errors, and began combining both technical and human factorstraining during flight simulator exercises. For example, a high-stressenvironment that requires clear, unequivocal communication between allof the crew members is the loss of an engine on take off. Although oftenregarded as a technical maneuver, it also requires accurate informationtransfer between the flight crew, the control tower, and the cabin crew.

Increasing the proficiency of flight crews with respect to human factorsknowledge and skills has come to be known as “cockpit resourcemanagement” (CRM) training. CRM training typically involves videotapinga flight simulator training exercise for later playback and review bythe flight crew during debriefing. Simulators equipped with videocameras can help flight crews sharpen their communication skills andother human factors skills in various emergency situations.Unfortunately, conventional video cassette recorder (VCR) technologypresently used to record flight simulator exercises is often inadequateduring debriefing. Rewinding and forwarding is often required in orderto locate a particular portion of an event recorded on videotape, and toreview a portion of the videotape multiple times. Some degree of timelag is typically involved because it is rather difficult to quickly jumpback and forth between different portions of the videotape. Anotherdisadvantage of using conventional VCR technology for capturing a flightsimulator training exercise for debriefing purposes is that it issomewhat difficult to simultaneously capture with video cameras, flightdata as represented by the various indicators and dials in thesimulator, and the actions of the flight crew.

U.S. Pat. No. 5,467,274 describes a method for debriefing aircraftoperations. As described, aircraft operations are recorded by both videorecorders and audio recorders and are stored by an airborne videotaperecorder (AVTR). The AVTR also records selected flight data as well asan associated time parameter. During debriefing operations, the varioussignals that have been recorded, such as video signals, audio signalsand other signals relating to flight parameters, are synchronized anddisplayed upon a three dimensional display.

Published PCT application No. WO 95/25313 (hereinafter the '313application) is not directed to a debriefing system or method but,instead, describes a method and apparatus for synchronizing andscheduling multiple datastreams, such as required in a variety ofmulti-media applications including computer-integrated telephony. Theapparatus of '313 application includes a selector that tracks the statusof each of the datastreams. In addition, a start pointer and an endpointer are associated with each datastream. Once the system time equalsthe start time of a respective datastream, the selector updates itsstatus to indicate if the start time has been reached. Each datastreamalso generally includes a trigger condition. Once the start time hasbeen reached, the selector therefore determines that the triggercondition has been met and, if so, forwards the respective datastream toa playout device. In order to synchronize multiple datastreams, thetrigger conditions of the datastreams may require that the start timesof two or more of the datastreams have been reached such that each ofthe datastreams is forwarded to the playout device at the same time.

SUMMARY OF THE INVENTION

It is therefore an object of the present invention to provide improveddebriefing systems and methods.

It is another object of the present invention to provide improveddebriefing systems and methods that overcome the limitations anddrawbacks of conventional VCR technology.

These and other objects are accomplished, according to the presentinvention, by event debriefing systems and methods for recording andreplaying various types of information generated during an event, bothsimulated and real. Facilities are provided for recording a plurality ofdatastreams related to an event, each datastream including chronologicalinformation related to at least one aspect of the event. Recordeddatastreams include video recordings of the actions of eventparticipants, and audio recording of audible communications of eventparticipants. Also included are instrumentation datastream from variousevent-monitoring instruments, including those necessary to createaircraft flight animation.

Facilities are provided whereupon, any point in time during an event canbe “indicated” with a time indication mark, created either by anobserver or automatically by the debriefing system. An observer of theevent may create time indication marks that facilitate recalling, duringdebriefing, what transpired at any particular point in time during theevent. The observer is presented with, preferably via a computerdisplay, menus of categories and sub-categories of pre-defined timeindication marks that can be selected and associated with the vent.Similarly, the debriefing system may be configured to automaticallycreate time indication marks when certain parameters are met during theevent.

Time indication marks may be defined for virtually any aspect of anevent including, but not limited to, categories related to the actionsof participants in the event (CRM for example), and categories relatedto technical aspects of the event. By associating a time during theevent with a particular category of time indication marks, selectivereplay of the various recorded datastreams is facilitated duringdebriefing. For example, if communications between event participants isgood (or bad) at a particular point in time during an event, apre-defined time indication mark may be selected from a menu andassociated with each of the recorded datastreams. During debriefing, theevent participants can view the datastreams in time synchronismbeginning at or near this time indication mark. Time indication marksthat are not pre-defined may also be created and associated with therecorded datastreams. For example, if something were to occur during anevent that would be worth reviewing during debriefing, yet does not fallwithin a particular pre-defined category, an observer can create a timeindication mark anyway.

The recorded datastreams are retrieved and presented in timesynchronism. In particular, the recorded datastreams can be retrievedand presented beginning at marked time indications and continuedchronologically thereafter in time synchronism. During debriefing, auser is presented with various computer screens from which particulartime indication marks can be selected. For example, a list of all markscreated during an event may be listed. Alternatively, various data froman event, with associated time indication marks, may be presented forselection to a user. By selecting a particular time indication mark, thevarious recorded datastreams are replayed in time synchronism beginningat or near the time associated with the selected time indication mark.

A display having a plurality of zones is used to present video andinstrumentation datastreams in time synchronism. Event-monitoringinstrumentation and other data is displayed in dynamic animation. Anaudio datastream is performed in time synchronism with the displayedvideo and instrumentation datastreams. Consequently, the disadvantagesof conventional VCR technology are overcome by the present invention.Any portion of a recorded event can be reviewed at the press of abutton, with virtually no time lag.

The present invention may be used in conjunction with aircraft flightsimulator debriefing systems and methods. Data, from both the flightsimulator and the video camera on the simulator flight deck, is directedto a computer. A computer screen is configured to display a variety ofpre-defined categories of time indication marks. A training facilitator,referred to as a “check pilot”, observes crew interactions during thesimulator exercise and selects time indication marks from the display.The time indication marks may be associated with various aspects of thesimulated flight, including technical and CRM aspects. For example, ifthe check pilot observes flight crew behavior during a simulated eventthat would be desirable to review during debriefing, he or she canselect a pre-defined time indication mark, or create a mark that is notpre-defined, and associate the mark with the recorded datastreams.

Software associated with the debriefing computer automaticallyreconstructs the simulator flight, links it to the video and audiosignal from the flight deck and displays the digitized video, dynamicanimation and selected instrumentation on the monitor located in adebriefing room. During debriefing following a simulator exercise, acheck pilot is able to go to any event during the flight and show thecrew exactly what transpired on the flight deck, complete with animatedinstrumentation and multiple views of the aircraft and flight path, allon a single monitor. Voice recognition software may also be includedwherein the conversations of the crew can be searched for selected wordsand phrases during the simulator exercise.

The present invention is advantageous because it substantially increasestraining efficiency and effectiveness by allowing the crew to “relive”their flight experience in a single integrated display of theirtechnical and human factor performance. Another advantage of thedebriefing system, according to the present invention, is that a flightcrew member can debrief himself or herself after a training exercise.This removes the subjectivity of the check pilot and puts him or her inthe position of having to simply manage the interaction between theflight crew and the simulator system. As an additional benefit, thepresent invention will result in increased standardization of checkpilot briefings, debriefings, and evaluations.

Prior to the present invention, marks on videotape lacked identifyinginformation. The check pilot knew that a mark existed, but there was notany descriptive information associated with the mark. Time indicationmarks, according to the present invention, facilitate the recall ofevents that transpired in the simulated exercise, particularly atcritical junctures, when the exercise is replayed during the debriefingsession. Because there is a tremendous amount of stimuli occurring (forexample, the flight instruments move dynamically, the visual scenesoutside of the cockpit move, and various audio communications takeplace, including bells, horns, communications between the flight crewand with air traffic control) during a simulated flight exercise, it isdifficult for the flight crew and the check pilot to recall all of theevents and data that occurred during a session. The cognitive processesof the flight crew and the check pilot cannot assimilate all the stimuliin order to effectively train motor skills and cognitive skills.

As described above, the invention implements systems and methods for:(1) recording, marking, and presenting multiple datastreams in timesynchronism; (2) recording, categorized marking, and presenting multipledatastreams in time synchronism; and (3) recording, automaticallymarking, and presenting multiple datastreams in time synchronism. Eachof these aspects may be used individually or in combination, to improvetraining efficiency and effectiveness during debriefing of an event.

The present invention is not limited to the recording and debriefing ofsimulated events. Many “real” events can be recorded and the actualevent participants debriefed using the present invention. For example,the systems and methods of the present invention can be utilized in thecontrol room of a nuclear power plant, the operating room of a hospital,the control center of the National Aeronautics and Space Administration(NASA) during a space mission, the assembly line of a manufacturingplant, and many other locations involving both technical aspects of anevent and interaction among participants in the event. The presentinvention is a powerful learning tool with many potential applications.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A schematically illustrates operations for recording, marking, andretrieving multiple datastreams, according to the present invention.

FIG. 1B schematically illustrates operations for marking an indicationof time during an event, according to the present invention.

FIG. 1C schematically illustrates operations for retrieving andpresenting multiple datastreams in time synchronism via time indicationmarks, according to the present invention.

FIG. 2 schematically illustrates a hardware configuration of the presentinvention when used in conjunction with a flight simulator.

FIG. 3 is a hierarchical menu of time indication marks when the presentinvention is used in conjunction with a flight simulator.

FIG. 4 illustrates an integrated debriefing computer display accordingto the present invention, when used on conjunction with a flightsimulator.

FIG. 5 illustrates a display screen for selecting a particular time markfrom a list of time indication marks during debriefing.

FIG. 6 illustrates an alternative display screen for selecting timeindication marks during debriefing.

FIG. 7 is a hierarchical menu of time indication marks according toanother aspect of the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The present invention now is described more fully hereinafter withreference to the accompanying drawings, in which preferred embodimentsof the invention are shown. This invention may, however, be embodied inmany different forms and should not be construed as limited to theembodiments set forth herein; rather, these embodiments are provided sothat this disclosure will be thorough and complete, and will fullyconvey the scope of the invention to those skilled in the art.

Referring now to FIG. 1A, debriefing operations of the present inventioninclude recording multiple datastreams related to an event (Block 100),marking an indication of time during the event (Block 200), andretrieving and presenting the recorded datastreams in time synchronisminstantly via the marked indication of time (Block 300). The multipledatastreams may include video, audio, event-monitoring, andinstrumentation datastreams. The debriefing system has the capability ofdigitizing and synchronizing multiple datastreams, displaying the datasimultaneously on a computer screen, and permitting instantaneous accessto information within each datastream for any particular point in time.

Referring now to FIG. 1B, operations for marking an indication of timeduring an event will now be described. An observer of an event canassociate a time indication mark with recorded information to facilitatereview during debriefing. A time indication mark is a field of datacontaining the category and sub-category headings and a time code for atime that the mark is created. Preferably, a plurality of pre-definedtime indication marks are organized by category and sub-category withina respective category (Blocks 202 and 204). An observer is presentedwith a computer display comprising a plurality of menus, windows, or thelike, from which a particular time indication mark can be selected(Block 206). A time code is associated with a selected time indicationmark (Block 208).

For example, in an event including multiple interacting participants, ifan observer witnesses participant behavior during the event that isparticularly good or bad, a pre-defined time indication mark may beselected from the computer display and a time code associated with theselected mark. During debriefing, the event participants can view thedatastreams in time synchronism beginning at or near the timerepresented by this time indication mark. Time indication marks that arenot pre-defined may also be created and associated with the recordeddatastreams. For example, if something were to occur during an eventthat would be worth reviewing during debriefing, yet does not fallwithin a particular pre-defined category, an observer is provided withthe ability to create a time indication mark. Additionally, timeindications marks may be automatically generated (Blocks 210 and 212).For example, certain events may trigger the creation of time indicationmarks, independent of an observer.

Referring now to FIG. 1C, operations for retrieving and presenting therecorded datastreams in time synchronism via a marked indication of timewill now be described. During debriefing of an event, a user ispresented, preferably via a computer display, a list of time indicationmarks created during the event (block 302). If the user selects a timeindication mark (Block 304), the multiple recorded datastreams aredisplayed in time synchronism at or near the point in time representedby the time indication mark (Block 306). Preferably, time indicationmarks are displayed by category. However, a variety of displayconfigurations may be used to facilitate the selection of a timeindication mark by a user.

The present invention permits various time indication marks to beassociated with the multiple datastreams whereupon the datastreams canbe accessed at these time indication marks during debriefing. Thepresent invention also permits the time indication marks to be selectedfrom a variety of categories, including technical aspects of therecorded event and human interaction aspects of the event participants.The invention is advantageous as a teaching tool for a variety ofsimulated events, especially these involving multiple participants,where risk and cost of the actual event are high. Exemplary events forwhich the debriefing system according to the present invention isadvantageous include, but are not limited to, aircraft operations,land-based vehicle operations including buses and train, surgicalprocedures, maritime simulations, and warfare simulations. The inventionis also advantageous as a teaching tool for a variety of real events,especially those involving technical aspects of the event and multipleparticipants. Exemplary real event locations include, but are notlimited to, power plant control rooms, hospital operating rooms, spaceagency control centers, war strategy rooms, operations centers ofairports, air traffic control facilities, television broadcast rooms,the bridge of a ship, sports events, negotiation/mediation rooms, andvarious industrial and manufacturing facilities.

In one embodiment, the present invention includes debriefing systems andmethods used in conjunction with an aircraft flight simulator. Flightcrew performance in a flight simulator exercise is generally measuredvia two components: technical proficiency and human factors or CRMproficiency. The technical component of flight crew performanceincludes, but is not limited to: knowledge and skills relating toaircraft systems; normal, abnormal and emergency operating procedures;air traffic control (ATC) procedures; instrument flight procedures; andnavigation and charts. The CRM component of flight crew performanceincludes, but is not limited to: communication skills; team building;the exercise of command and use of leadership; decision making; andmanagement of resources, workload and stress.

The present invention comprises software, hardware, and interfacesnecessary for capturing multiple datastreams from the flight simulator,and from video cameras and microphones in the simulator cockpit. Datafrom the flight simulator, and from video cameras and microphones in thesimulator cockpit. Data from the flight simulator includes flight andengine instrumentation data, and other data required to create animatedinstrumentation and aircraft position displays. Typically, thesedatastreams are generated in digital format. On the other hand, theaudio and video datastreams are typically generated in analog format.The present invention may convert analog datastreams into digitaldatastreams. Furthermore, the present invention can synchronize andstore the multiple datastreams. An important aspect of the presentinvention is the ability to create various time indication marks thatcan be associated with the recorded datastreams. The time indicationmarks permit virtually instant access to the multiple datastreams duringdebriefing.

During a typical flight simulator debriefing session following asimulated flight exercise, the present invention displays the digitizedvideo datastream in time synchronism with dynamic animation of flightinstrumentation and plane position in a series of windows on a computermonitor. Audio playback, also in time synchronism with the displayedvideo and animated datastreams, is provided through a speaker system.Thus, plane position, instrument readings, and crew actions/interactionsat any point in time can be viewed easily. During playback, the varioussynchronized datastreams can be fast forwarded, and reversed tovirtually any point in time during the flight exercise with almost nodelay.

Referring now to FIG. 2, a preferred hardware configuration of thepresent invention, when utilized as a debriefing system in conjunctionwith a flight simulator, is schematically illustrated. In general, thecomponents of the debriefing system 10 include a flight simulator cab20, a flight simulator host computer 30, a check pilot interface 40, amarking system 50, audio and video recording system 60, a debriefingcomputer 70, and a user interface 80 for the debriefing computer. As isknown by those having skill in the art, a flight crew undergoes a flighttraining exercise within the flight simulator cab 20, which iscontrolled by the flight simulator host computer 30. The instrumentationwithin the flight simulator cab 20, which is typically a reproduction ofthe cockpit of a particular aircraft, is under computer control via thehost computer 30. Control over a training exercise is provided to acheck pilot via the check pilot interface 40.

In the illustrated embodiment, the flight simulator cab 20 is fittedwith a plurality of microphones 22 for recording the audiblecommunications of flight crew members. In a typical configuration, four(4) cockpit microphones are used in conjunction with a cockpit-wide areamicrophone, Captain and First Officer boom microphones, andpanel-mounted microphones. An audio mixer 23 combines the audio inputfrom each microphone into a single audio source. The actions of theflight crew are recorded via one or more video cameras 24 located withinthe flight simulator cab 20. Preferably, each video camera 24 is aninfrared low light camera. A video and audio booster 25 boosts the videoand audio analog datastreams over long distances between the recordingsystem 60 and the debriefing room computer 70, thereby facilitating thesubsequent digitization of these datastreams. A hum buckler 26 ispreferably provided to minimize audio and video datastream noise overlong distances between the recording system 60 and the debriefing roomcomputer 70. However, the present invention is not limited to analogvideo and audio recording devices. Digital recording devices, includingcharge coupled device (CCD) cameras, may be utilized.

The present invention incorporates critical flight information withdigitized audio and video obtained directly from the simulator. Thisinformation is instantly reproduced in a configurable display of thedebriefing room computer 70. The monitor preferably provides a chaseplane view from any vantage point which enhances situation awarenessduring debriefing. Preferably, the debriefing room computer 70 is aSilicon Graphics Model R4000, or equivalent, having: audio and videocapture and playback boards; serial or Ethernet communications ports; akeyboard, a monitor suitable for viewing by small groups, a mouse, anddata storage capacity of at least four (4) gigabytes. Even morepreferable is data storage of at least eight (8) gigabytes. Software ofthe debriefing room computer 70 performs the following tasks: captureand store video and audio datastreams; store flight instrumentationdatastreams and other datastreams from the simulator; store data fromthe marking system personal computer; replay audio and videodatastreams; and replay flight simulator data in 4D animation. 4Danimation provides the ability to view three dimensional (3D) animationfrom any point in space. Preferably, the movement of a computer mouse toany point around an object will allow a user to observe the object fromthat point in space. In addition, software is provided that makes the 4Danimator speed up or slow down the frame rate replay to keep insynchronism with the video replay.

Software is also provided that decodes simulator flight instrumentindications into exceedances. The term exceedance refers to somethingthat happened that should not have happened. For example, if an aircraftexceeds a maximum speed, this would be an exceedance. The maximum speedfor an aircraft is stored in a computer file. The maximum speed may befurther qualified by condition sets, such as maximum speed at aparticular altitude. The program constantly compares the speed of thesimulated aircraft with the file to determine if the maximum speed hasbeen exceeded. If it has been exceeded, a mark and exceedance iscreated.

Control over the debriefing computer 70 is maintained through amouse-driven user interface, preferably having various menus for startup and configuration. In addition, it is preferred that the userinterface have a panel similar to the protocol used by a VCR interface,thereby facilitating ease of use. For example, “fast forward”, “rewind”,“play”, “pause”, and “stop” buttons are displayed on the screen.Preferably, buttons are provided for jumping to the previous or nexttime indication mark. The user can control playback during debriefingsimply by clicking on the appropriate button displayed on the screen.All windows on the debriefing computer monitor are synchronized toremain in balance with the video during playback. Audio is providedthrough a speaker system linked with the debriefing computer and isfully synchronized with the video and graphics display.

Referring now to FIG. 4, a preferred embodiment of the debriefingcomputer display 72, when used in conjunction with a flight simulator,is illustrated. Preferably, the display is a thirty-five inch (35″)color monitor with at least four (4) different windows. A fifth windowmay be provided for displaying a list of time indication marks. Theconfigurable display windows can be enlarged, moved, and reduced asdesired. In the illustrated configuration, video display window 73displays a digital video recording of a flight crew during a simulatorexercise. Plane position display window 74 illustrates the position ofan airplane from a variety of view points at any stage during asimulated flight exercise. In addition, ideal plane positions may besuperimposed over actual plane positions in the plane position displaywindow 74. The plane position window 74 also may display pertinentinformation such as glide slope and weather features. Instrumentationdisplay windows 75 contain animated displays of selected aircraftinstrumentation, configurable depending on the type of aircraft being“flown” in the simulator, and the particular simulated flight exerciseinvolved. The animated instrumentation is a precise reproduction of theappearance of the actual instrumentation in the cockpit of thesimulator. The animation is dynamic in that the various instrumentsdisplayed appear to present data in real time. For example, the groundspeed indicator and altimeter illustrated in FIG. 4 give continuousreadings during playback of a particular portion of the simulatedexercise.

All display windows are synchronized such that the dynamicinstrumentation and plane position animation, and digital video andaudio of flight crew actions, represent what transpired at a particularpoint in time during the recorded exercise. Because all displayed videoand instrumentation data is in digital format, any point in time duringthe recorded flight exercise can be accessed instantly. In addition, asdescribed in greater detail hereinafter, time indication marks can becreated, both manually by the check pilot, automatically by anexceedance routine, and automatically by the simulator, at variouspoints in time during the recording of the flight exercise so that,during debriefing, these particular points in time can be accessedinstantly.

Each time indication mark facilitates quick access to what transpired atthat particular point in time during the simulated flight exercise whenthe recorded datastreams are played back during debriefing. Duringdebriefing, the flight crew can observe their actions, listed to theircommunications, observe flight instrumentation and plane position priorto, simultaneously with, and subsequent to any time mark. For example,the recorded datastreams can be replayed beginning ten (10) secondsprior to a given time mark and continued therefrom. A plethora of replayoptions are made possible by the present invention, and are not limitedto those described herein. The many possible replay options create avaluable tool for event participants to use in learning from an actualor simulated event.

The marking system 50 permits a check pilot to create time indicationmarks during a simulated flight exercise and to associate one or morecategories of performance factors with the time marks. Preferably, themarking system 50 is controlled by a personal computer having a “touchscreen” display, and is located in the flight simulator cab 20. Themarking system user interface presented on the touch screen is describedin detail below and is designed to facilitate ease of use by the checkpilot. However, non-touch screen computer monitors may be utilized aswell to present the marking system user interface. Conventionalselection techniques including keyboard and mouse functions may also beused to create time indication marks.

The marking system 50 includes software for producing a main menu andhierarchical sub-menus on the marking system display and for sendingtime mark data to the debriefing room computer 70. Preferably, themarking system 50 is connected to the debriefing room computer 70 via anEthernet link or a serial port cable. In addition, the marking system 50provides the check pilot with the ability to control the recording ofthe various video, audio, and instrumentation datastreams by thedebriefing room computer 70. The marking system 50 instructs thedebriefing system when to start, stop, and mark.

During a flight simulation exercise, the check pilot is presented with amain menu 52, as shown at the top of FIG. 3, on the marking systemdisplay that includes the following categories of time indication marks:Human Factors, Technical, LOFT Event Trigger, Auto Detect, and End. Fromthis main menu 52, the check pilot is presented with sub-categories 54within each category, as illustrated in FIG. 3. Each sub-categorypreferably contains a plurality of pre-defined time indication marksthat can be selected as described below. Pre-defined time indicationmarks and their respective categories and sub-categories may bepresented to the user in a variety of ways, and are not limited tomenus. Other methods of presenting categories and sub-categories of timeindication marks to a user include using separate windows or icons.

A time indication mark is defined as a field of data containing the maincategory heading, the sub-category heading(s), and a time code index atthe time that the mark is received on the debriefing room computer 70.After each mark is selected, the time indication mark data is stored onthe debriefing room computer 70, and the marking system 50 returns tothe main menu of time mark categories, ready for the next time mark.

Sub-categories of the Human Factors marks include, but are not limitedto: “Briefings”, “Inquiry/Assertion/Advocacy”, “Crew Self Critique”,“Communications/Decisions”, “Leadership/Followership”, “InterpersonalRelations”, “Preparation/Planning/Vigilance”, and “Workload Management.”These sub-categories are recommended by the Federal AviationAdministration (FAA) in the Crew Resource Management Advisory Circular(AC 120-51), which is incorporated herein by reference in tis entirety.These sub-categories are CRM issues that airline training departmentsaround the world have adopted as criteria, goals, and models towardswhich they strive to attain proficiency in. Tables 1-8 below set forthillustrative pre-defined time indication marks for each of theabove-listed human factors sub-categories.

TABLE 1 Briefings The effective briefing is interesting and thorough. Itaddresses coordination, planning, and problems. Although briefings areprimarily a captain's responsibility, other crew members may addsignificantly to planning and should be encouraged to do so. TimeIndication Marks and Descriptions The briefing establishes anenvironment for open/interactive communications (for example, thecaptain calls for questions or comments, answers questions directly,listens with patience, does not interrupt or “talk over,” does not rushthrough the briefing, and makes eye contact as appropriate). Thebriefing is interactive and emphasizes the importance of questions,critique, and the offering of information. The briefing establishes a“team concept” (for example, the captain uses “we” language, encouragesall to participate and to help with the flight). The briefing coverspertinent safety and operational issues. The briefing identifiespotential problems such as weather, delays, and abnormal systemoperations. The briefing provides guidelines for crew actions; divisionof labor and crew workload is addressed. The briefing includes the cabincrew as part of the team. The briefing sets expectations for handlingdeviations from standard operating procedures. The briefing establishesguidelines for the operation of automated systems (for example, whensystems will be disabled; which programming actions must be verbalizedand acknowledged). The briefing specifies pilot flying and pilot notflying duties and responsibilities with regard to automated systems.

TABLE 2 Inquiry/Advocacy/Assertion These behaviors relate tocrewmembers' promoting the course of action that they feel is best, evenwhen it involves conflict with others. Time Indication Marks andDescriptions Crewmembers speak up and state their information withappropriate persistence until there is some clear resolution. “Challengeand response” environment is developed. Questions are encouraged and areanswered openly and nondefensively. Crewmembers are encouraged toquestion the actions and decisions of others. Crewmembers seek help fromothers when necessary. Crewmembers question status and programming ofautomated systems to confirm situational awareness.

TABLE 3 Crew Self-Critique Regarding Decisions and Actions Thesebehaviors relate to the effectiveness of a group and/or an individualcrewmember in critique and debriefing. Areas covered should include theproduct, the process, and the people involved. Critique may occur duringan activity, and/or after completing it. Time Indication Marks andDescriptions Critique occurs at appropriate times, which may be times oflow or high workload. Critique deals with positive as well as negativeaspects of crew performance. Critique involves the whole crewinteractively. Critique makes a positive learning experience. Feedbackis specific, objective, usable, and constructively given Critique isaccepted objectively and nondefensively.

TABLE 4 Communications/Decisions These behaviors relate to free and opencommunication. They reflect the extent to which crewmembers providenecessary information at the appropriate time (for example, initiatingchecklists and alerting others to developing problems). Activeparticipation in the decision making process is encouraged. Decisionsare clearly communicated and acknowledged. Questioning of actions anddecisions is considered routine. Time Indication Marks and DescriptionsOperational decisions are clearly stated to other crewmembers.Crewmembers acknowledge their understanding of decisions. “Bottom lines”for safety are established and communicated. The “big picture” and thegame plan are shared within the team, including flight attendants andothers as appropriate. Crewmembers are encouraged to state their ownideas, opinions, and recommendations. Efforts are made to provide anatmosphere that invites open and free communications. Initial entriesand changed entries to automated systems are verbalized andacknowledged.

TABLE 5 Leadership Followership/Concern for Tasks These behaviors relateto appropriate leadership and followership. They reflect the extent towhich the crew is concerned with the effective accomplishment of tasks.Time Indication Marks and Descriptions All available resources are usedto accomplish the job at hand. Flight deck activities are coordinated toestablish an acceptable balance between respect for authority and theappropriate practice of assertiveness. Actions are decisive when thesituation requires. A desire to achieve the most effective operationpossible is clearly demonstrated. The need to adhere to standardoperating practices is recognized. Group climate appropriate to theoperational situation is continually monitored and adjusted (forexample, social conversation may occur during low workload, but nothigh). Effects of stress and fatigue on performance are recognized. Timeavailable for the task is well managed. Demands on resources posed byoperation of automated systems are recognized and managed. Whenprogramming demands could reduce situational awareness or create workoverloads, levels of automation are reduced appropriately.

TABLE 6 Interpersonal Relationships/Group Climate These behaviors relateto the quality of interpersonal relationships and the pervasive climateof the flight deck. Time Indication Marks and Descriptions Crewmembersremain calm under stressful conditions. Crewmembers show sensitivity andability to adapt to the personalities of others. Crewmembers recognizesymptoms of psychological stress and fatigue in self and in others (forexample, recognizes when he/she is experiencing “tunnel vision” andseeks help from the team; or notes when a crewmember is notcommunicating and draws him/her back into the team). “Tone” in thecockpit is friendly, relaxed, and supportive. During times of lowcommunication, crewmembers check in with others to see how they aredoing.

TABLE 7 Preparation/Planning/Vigilance These behaviors relate to crews'anticipating contingencies and the various actions that may be required.Excellent crews are always “ahead of the curve” and generally seemrelaxed. They devote appropriate attention to required tasks and respondwithout undue delay to new developments. (They may engage in casualsocial conversation during periods of low workload and not necessarilydiminish their vigilance.) Time Indication Marks and DescriptionsDemonstrating and expressing situational awareness; for example, the“model” of what is happening is shared within the crew. Activemonitoring of all instruments and communications and sharing relevantinformation with the rest of the crew. Monitoring weather and trafficand sharing relevant information with the rest of the crew. Avoiding“tunnel vision” caused by stress; for example, stating or asking for the“big picture”. Being aware of factors such as stress that can degradevigilance and watching for performance degradation in other crewmembers.Staying “ahead of the curve” in preparing for planned situations orcontingencies. Ensuring that cockpit and cabin crewmembers are aware ofplans. Including all appropriate crewmembers in the planning process.Allowing enough time before maneuvers for programming of the flightmanagement computer. Ensuring that all crewmembers are aware of initialentries and changed entries in the flight management system.

TABLE 8 Workload Distributed/Distractions Avoided These behaviors relateto time and workload management. They reflect how well the crew managesto prioritize tasks, share the workload, and avoid being distracted fromessential activities. Time Indication Marks and Descriptions Crewmembersspeak up when they recognize work overloads in themselves or in others.Tasks are distributed in ways that maximize efficiency. Workloaddistribution is clearly communicated and acknowledged. Nonoperationalfactors such as social interaction are not allowed to interfere withduties. Task priorities are clearly communicated. Secondary operationaltasks (for example, dealing with passenger needs and communications withcompany) are prioritized so as to allow sufficient resources for primaryflight duties. Potential distractions posed by automated systems areanticipated, and appropriate preventive action is taken, includingreducing or disengaging automated features as appropriate.

Should the check pilot in the simulator observe either positive ornegative behavior in these sub-categories, he or she will indicate thatobservation by creating one of the above, or other, time indicationmarks. Preferably, the creation of the time indication mark isaccomplished by touching one of these listed time indication marks onthe touch screen. However, a time indication mark may be created in waysother than via a touch screen, such as via a mouse or keyboard function.In addition, time indication marks may be created that are notpre-defined within a particular category or sub-category. For example,if something were to occur during a simulated event that would be worthreviewing during debriefing, yet does not fall within a particularpre-defined category, a check pilot can create a generic time indicationmark to mark this particular point in time during the simulated event.

Technical sub-categories of time indication marks include the minimumnumber of maneuvers that a flight crew should be able to performproficiently. These sub-categories may vary from one country to another,but the majority of them are standard among airlines of the world.Generally, these sub-categories include, but are not limited to:“Rejected Takeoff”, “V1 Cut”, “Area Departure”, “Area Arrival”, NonPrecision Approaches”, “Holding”, “ILS Approaches”, “Missed Approaches”,“Landings”, and “Windshear.” As shown, various sub-categories of thesesub-categories exist. The check pilot in the simulator may select timeindication marks from any of the technical sub-categories.

“LOFT” is the acronym from “Line Oriented Flight Training.” The airlineindustry has been moving away from proficiency-based maneuvers of atechnical nature and more towards simulations that are closely orientedtowards actual line operations. The term “line operations” refers toflying with passengers. The emphasis is on human factors training andevaluation, rather than on motor skills, training, demonstration, andevaluation. A “LOFT” event trigger is defined as an external event thatis introduced into a simulated flight exercise and that is specificallydesigned to induce a human factors event. For example, an engine oiltemperature gauge may show an erratic reading during a simulated flightto distract the flight crew. The distraction may lead to failure ofworkload management or communications between the flight crew. The focusduring debriefing would be to manage the distraction so that the flightmay successfully continue notwithstanding the distraction. The checkpilot in the simulator may select time indication marks from any LOFTcategories and sub-categories.

“Auto Detect” includes various automatically-created time indicationmark sub-categories. Exemplary sub-categories include: when the checkpilot initiates certain simulator functions, and when certainperformance parameters are exceeded. For example, a time indication markmay be automatically created when the check pilot requests the simulatorto induce windshear. As another example, a time indication mark may beautomatically created when the aircraft is operated in excess of aparticular airspeed at a particular altitude. Automatically created timeindication marks can be numerous and it is expected that users willcustomize the present invention to fit particular needs.

Referring now to FIG. 5, a debriefing room computer display screen 82for selecting a particular time indication mark from a list of timeindication marks 83 during debriefing is illustrated. A user enters thenumber of the time indication mark in the field labeled “Enter Mark #”and presses a keyboard or mouse function to begin playback of thevarious datastreams at the point in time indicated by the selected mark.The screen 82 permits the user to view additional screen listings oftime marks through “next page” and “previous page” functions and theirequivalents. Alternatively, a display of elapsed time at a particularmark may be presented along with controls for playback from a selectedelapsed time.

As shown, a time code 84, representing a point in time during aparticular recorded flight exercise, is associated with each timeindication mark. In addition, both a mark category 85 and sub-category86 are associated with each time indication mark. For example, as shownin FIG. 5, at a time equal to ten (10) seconds into a particular flightexercise, a time mark was created for a “V1 Cut” technical maneuver.Upon selecting this mark during debriefing, the display illustrated inFIG. 4 would replay, in digital format and in time synchronism, thevideo of the flight crew, the plane position, and various flightinstrumentation beginning at ten (10) seconds into the flight exercise.However, it is to be understood that the present invention permitsreplay to be configurable whereupon it can begin and end before, after,or precisely at the ten (10) second mark.

Referring now to FIG. 6, another debriefing room computer display screen88 for selecting time indication marks during debriefing is illustrated.The illustrated screen displays a plot 91 of aircraft altitude during aparticular segment of a flight exercise. Altitude in feet is plottedalong the “y” axis 89, and time is plotted along the “x” axis 90.Vertical marks 92 represent time indication marks created via themarking system 50 during a simulated flight exercise. During thesimulated flight exercise, the marks were either automatically createdby the simulator, or by the check pilot as described previously.

During debriefing, a user selects a particular mark by “clicking” on themark with a mouse. When selected, the details of what happened at thatpoint in time can be displayed on the debriefing room computer monitoras previously described above. In addition, when a mark is selected,preferably a window containing the data for the selected time indicationmark is displayed. Preferably, the time marks are color-coded based onthe type of mark to facilitate easy recognition. For example, some orall of the vertical marks 92 in FIG. 6 may be magenta to representTechnical marks. Some or all of the vertical marks 92 in FIG. 6 may bered to represent Human Factors marks. Some or all of the vertical marks92 in FIG. 6 may be yellow may to represent LFOT Event Trigger marks.Some or all of the vertical marks 92 in FIG. 6 may be white to representAuto Detect marks. In addition, the screen has zoom-in and zoom-outcapability, to facilitate easy use should an area of the screen becomecluttered with marks.

Referring now to FIG. 7, the main menu 52 on the marking system displaymay include a category for voice recognition. Audio datastreams producedby recording the audible communications of flight crew members may beanalyzed using voice recognition software. As is known to those havingskill in the art, words, phrases and other portions of an audiodatastream may be selectively located using the voice recognitionsoftware. Preferably, keyword searching is provided to facilitatelocating selective portions of an audio datastream. For example, asearch could be performed during debriefing to determine whether thecrew used the word “abort” or “fire” during the simulated exercise.Selectively located words and other portions of an audio datastream mayserve as time indication marks and may be stored for later retrieval.

According to another aspect of the present invention, video examples ofappropriate flight crew behavior, may be instantly available for aparticular event during a simulator exercise. For example, if a CRMerror occurs during a session, the system, if prompted during debriefingor briefing will respond with a video example of crewactions/interactions that would be appropriate for the particular event.

The systems and methods according to the present invention can be usedas a teaching tool for a variety of events. For example, during asurgical procedure, a video camera can be used to record the event. Thesystem can be configured to interface with the video camera as well aswith instrumentation used during the procedure, such as monitors,respirators, and the like. The video and/or audio signal is digitizedand synchronized with the instrumentation data, and then stored.Subsequently, the stored video and instrumentation data can bedisplayed, thereby showing the actions/interactions of a surgical teamsimultaneously with various instrumentation readings. The end result isa valuable teaching aid for use in debriefing the surgical team and fortraining others.

The present invention is a vast improvement over existing tools fordebriefing training exercises and real events. The present inventionincreases training efficiency and effectiveness by allowing trainees to“re-live” their training or operations experience via a singleintegrated display of their performance with respect to both technicalaspects and human factors aspects.

The foregoing is illustrative of the present invention and is not to beconstrued as limiting thereof. Although a few exemplary embodiments ofthis invention have been described, those skilled in the art willreadily appreciate that many modifications are possible in the exemplaryembodiments without materially departing from the novel teachings andadvantages of this invention. Accordingly, all such modifications areintended to be included within the scope of this invention as defined inthe claims. In the claims, means-plus-function clauses are intended tocover the structures described herein as performing the recited functionand not only structural equivalents but also equivalent structures.Therefore, it is to be understood that the foregoing is illustrative ofthe present invention and is not to be construed as limited to thespecific embodiments disclosed, and that modifications to the disclosedembodiments, as well as other embodiments, are intended to be includedwithin the scope of the appended claims. The invention is defined by thefollowing claims, with equivalents of the claims to be included therein.

That which is claimed:
 1. An event debriefing system (10) comprisingmeans (60) for recording a plurality of datastreams related to an eventwhich occurs over an extended time period, each datastream includingchronological information related to at least one aspect of the eventover the extended time period, wherein the event debriefing system (10)is characterized by and further comprises: categorized marking means(50), operating concurrent with said recording means (60), for markingdiscrete times during the extended time period at which items ofinterest occur, and for associating a category with each of the markedtimes based upon characteristics of the respective item; and presentingmeans (70), responsive to said marking means (50) and to the selectionof a respective category, for retrieving information from said recordeddatastreams based upon the marked times in the selected category, andfor presenting the retrieved information from the recorded datastreamssuch that those portions of the event associated with the selectedcategory are presented.
 2. An event debriefing system (10) according toclaim 1 further comprising an aircraft flight simulator (20) forsimulating aircraft flight such that the event is a simulated aircraftflight.
 3. An event debriefing system (10) according to claim 1 for usein simulating land-based vehicle operations.
 4. An event debriefingsystem (10) according to claim 1 for use in monitoring a manufacturingfacility.
 5. An event debriefing system (10) according to claim 1,wherein said associated category includes a category related to actionsof human participants in said event.
 6. An event debriefing system (10)according to claim 1, wherein said associated category includes acategory related to technical parameters of said event.
 7. An eventdebriefing system (10) according to claim 1, wherein said synchronizedpresenting means (70) comprises means for retrieving information fromsaid datastreams beginning at the marked time and continuingchronologically thereafter, and for presenting the retrieved informationfrom the datastreams in time synchronism.
 8. An event debriefing system(10) according to claim 1, wherein said recording means (60) comprises:video recording means (24) for producing a video datastream of actionsof event participants; audio recording means (24) for producing an audiodatastream of audible communications of event participants; andinstrumentation recording means for producing an instrumentationdatastream from various event monitoring instruments.
 9. An eventdebriefing system (10) according to claim 8 wherein said synchronizedpresenting means (70) comprises a display (72) having a plurality ofzones, and wherein said video and instrumentation datastreams arepresented in a respective one of said plurality of zones in timesynchronism.
 10. An event debriefing system (10) according to claim 9wherein said synchronized presenting means (70) comprises means forperforming said audio datastream with said displayed video andinstrumentation datastreams in time synchronism.
 11. An event debriefingsystem (10) according to claim 10 wherein said performing means furthercomprises voice recognition means for selectively locating portions ofsaid audio datastream.
 12. An event debriefing system (10) according toclaim 1 wherein said marking means (50) is responsive to useridentification of a time during said extended time period.
 13. An eventdebriefing system (10) according to claim 1, wherein said synchronizedpresenting means (70) includes means (75) for displaying the appearanceof selected instrumentation.
 14. An event debriefing system (10)comprising means (60) for recording a plurality of datastreams relatedto an event which occurs over an extended time period, each datastreamincluding chronological information related to at least one aspect ofthe event over the extended time period, wherein the event debriefingsystem (10) is characterized by and further comprises: automatic markingmeans (50), for automatically marking a time during the extended timeperiod at which at least one of said aspects meets predeterminedcriteria, and presenting means (70), responsive to said automaticmarking means, for retrieving information from said recorded datastreamsbased upon the automatically marked time, and for presenting theretrieved information from the recorded datastreams such that a portionof the event during which at least one of the aspects meetspredetermined criteria is displayed.
 15. An event debriefing system (10)according to claim 14 further comprising an aircraft flight simulator(20) for simulating aircraft flight such that the event is a simulatedaircraft flight.
 16. An event debriefing system (10) according to claim14 for use in simulating land-based vehicle operations.
 17. An eventdebriefing system (10) according to claim 14 for use in monitoring amanufacturing facility.
 18. An event debriefing system (10) according toclaim 14 wherein said synchronized presenting means (70) comprises meansfor retrieving information from said datastreams beginning at the markedtime and continuing chronologically thereafter, and for presenting theretrieved information from the datastreams in time synchronism.
 19. Anevent debriefing system (10) according to claim 14 wherein saidrecording means (60) comprises: video recording means (24) for producinga video datastream of actions of event participants; audio recordingmeans (24) for producing an audio datastream of audible communicationsof event participants; and instrumentation recording means for producingan instrumentation datastream from various event monitoring instruments.20. An event debriefing system (10) according to claim 19 wherein saidsynchronized presenting means (70) comprises a display (72) having aplurality of zones, and wherein said video and instrumentationdatastreams are presented in a respective one of said plurality of zonesin time synchronism.
 21. An event debriefing system (10) according toclaim 20 wherein said synchronized presenting means (70) comprises meansfor performing said audio datastream with said displayed video andinstrumentation datastreams in time synchronism.
 22. An event debriefingsystem (10) according to claim 14 further comprising categorized markingmeans (50) for marking time during the extended time period and forassociating a category with the marked times.
 23. An event debriefingsystem (10) according to claim 14, wherein said synchronized presentingmeans (70) includes means (75) for displaying the appearance of selectedinstrumentation.
 24. A method of event debriefing comprising recording(100) a plurality of datastreams related to an event which occurs overan extended time period, each datastream including chronologicalinformation related to at least one aspect of the event over theextended time period, wherein the method of event debriefing ischaracterized by the further steps of: identifying discrete times duringthe event at which items of interest occur; marking (200) those discretetimes during said step of recording the plurality of datastreams atwhich items of interest occur; and retrieving (300) information fromsaid recorded datastreams based upon the marked times, and presentingthe retrieved information from the datastreams in time synchronism suchthat selected portions of the event are displayed.
 25. A method of eventdebriefing according to claim 24 wherein said marking step (200)comprises automatically marking (212) those discrete times during saidstep (100) of recording the plurality of datastreams at which at leastone of the aspects meets predetermined criteria such that those portionsof the event during which at least one of the aspects meetspredetermined criteria can be displayed.
 26. A method of eventdebriefing according to claim 24, wherein said step (200) of markingtimes comprises: marking discrete times during the extended time periodat which items of interest occur; associating (208) a category with eachof the marked times based upon characteristics of the respective item;and retrieving information from said recorded datastreams based upon themarked times in a selected category, and presenting the retrievedinformation from the recorded datastreams such that those portions ofthe event associated with the selected category are presented.
 27. Amethod of event debriefing according to claim 26, wherein said step(208) of associating a category includes associating a category relatedto actions of human participants in said event with the marked times.28. A method of event debriefing according to claim 26, wherein saidstep (208) of associating a category includes associating a categoryrelated to technical parameters of said event.
 29. A method of eventdebriefing according to claim 24 further comprising a step of simulatingaircraft flight such that the event is a simulated aircraft flight. 30.A method of event debriefing according to claim 24 for use in simulatingland-based vehicle operations.
 31. A method of event debriefingaccording to claim 24 for use in monitoring a manufacturing facility.32. A method of event debriefing according to claim 24 wherein said step(300) of retrieving information from said datastreams based upon themarked time comprises retrieving information from said datastreamsbeginning at the marked time and continuing chronologically thereafter,and presenting (306) the retrieved information from the datastreams intime synchronism.
 33. A method of event debriefing according to claim 24wherein said step (100) of recording a plurality of datastreams relatedto an event comprises: producing a video datastream of actions of eventparticipants; producing an audio datastream of audible communications ofevent participants; and producing an instrumentation datastream fromvarious event monitoring instruments.
 34. A method of event debriefing,according to claim 33 wherein said step (300) of retrieving andpresenting the retrieved information comprises displaying said video andinstrumentation datastreams in a respective one of a plurality ofdisplay zones in time synchronism.
 35. A method of event debriefingaccording to claim 34, further comprising the step of performing saidaudio datastream with said displayed video and instrumentationdatastreams in time synchronism.
 36. A method of event debriefingaccording to claim 35 wherein said performing step further comprises thestep of selectively locating voice portions of said audio datastream.